{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hill Climbing (with adaptive noise scaling, without tensors) \n",
    "# CartPole-v1\n",
    "\n",
    "---\n",
    "\n",
    "   In this notebook, we train the Hill Climbing Agent with   \n",
    "**adaptive noise scaling** with OpenAI Gym's Cartpole environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Policy-Based Method\n",
    "\n",
    "In **policy-based methods**, instead of learning a **value function** that tells us what is     \n",
    "the expected sum of rewards given a state and an action, we learn directly the **policy function**    \n",
    "that maps state to action (select actions without using a value function)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Import the Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "#%matplotlib inline\n",
    "import time\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Define the Policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "observation space: Box(4,)\n",
      "action space: Discrete(2)\n",
      "threshold:  475.0\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CartPole-v1')\n",
    "print('observation space:', env.observation_space)\n",
    "print('action space:', env.action_space)\n",
    "threshold = env.spec.reward_threshold\n",
    "print('threshold: ', threshold)\n",
    "\n",
    "class Policy():\n",
    "    def __init__(self, s_size=4, a_size=2):\n",
    "        self.w = 1e-4*np.random.rand(s_size, a_size)  # weights for simple linear policy: state_space x action_space\n",
    "        \n",
    "    def forward(self, state):\n",
    "        x = np.dot(state, self.w)\n",
    "        return np.exp(x)/sum(np.exp(x))\n",
    "    \n",
    "    def act(self, state):\n",
    "        probs = self.forward(state)\n",
    "        #action = np.random.choice(2, p=probs) # option 1: stochastic policy\n",
    "        action = np.argmax(probs)              # option 2: deterministic policy\n",
    "        return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent with Stochastic Policy Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep.:   1 , timesteps: 192, noise_scale (Gt >= best(Gt)): 0.0050, Gt 85.4803, \tAvg.Score:  192.000\n",
      "Ep.:   2 , timesteps:  54, noise_scale (Gt < best(Gt)): 0.0100, Gt 41.8834, \tAvg.Score:  123.000\n",
      "Ep.:   3 , timesteps:  82, noise_scale (Gt < best(Gt)): 0.0200, Gt 56.1382, \tAvg.Score:  109.333\n",
      "Ep.:   4 , timesteps:  11, noise_scale (Gt < best(Gt)): 0.0400, Gt 10.4662, \tAvg.Score:  84.750\n",
      "Ep.:   5 , timesteps:   8, noise_scale (Gt < best(Gt)): 0.0800, Gt 7.7255, \tAvg.Score:  69.400\n",
      "Ep.:   6 , timesteps:  32, noise_scale (Gt < best(Gt)): 0.1600, Gt 27.5020, \tAvg.Score:  63.167\n",
      "Ep.:   7 , timesteps:  10, noise_scale (Gt < best(Gt)): 0.3200, Gt 9.5618, \tAvg.Score:  55.571\n",
      "Ep.:   8 , timesteps:   9, noise_scale (Gt < best(Gt)): 0.6400, Gt 8.6483, \tAvg.Score:  49.750\n",
      "Ep.:   9 , timesteps:  10, noise_scale (Gt < best(Gt)): 1.2800, Gt 9.5618, \tAvg.Score:  45.333\n",
      "Ep.:  10 , timesteps:  34, noise_scale (Gt < best(Gt)): 2.0000, Gt 28.9447, \tAvg.Score:  44.200\n",
      "Ep.:  11 , timesteps:   8, noise_scale (Gt < best(Gt)): 2.0000, Gt 7.7255, \tAvg.Score:  40.909\n",
      "Ep.:  12 , timesteps:  11, noise_scale (Gt < best(Gt)): 2.0000, Gt 10.4662, \tAvg.Score:  38.417\n",
      "Ep.:  13 , timesteps:  42, noise_scale (Gt < best(Gt)): 2.0000, Gt 34.4341, \tAvg.Score:  38.692\n",
      "Ep.:  14 , timesteps: 500, noise_scale (Gt >= best(Gt)): 1.0000, Gt 99.3430, \tAvg.Score:  71.643\n",
      "Ep.:  15 , timesteps:  95, noise_scale (Gt < best(Gt)): 2.0000, Gt 61.5104, \tAvg.Score:  73.200\n",
      "Ep.:  16 , timesteps: 103, noise_scale (Gt < best(Gt)): 2.0000, Gt 64.4839, \tAvg.Score:  75.062\n",
      "Ep.:  17 , timesteps: 155, noise_scale (Gt < best(Gt)): 2.0000, Gt 78.9402, \tAvg.Score:  79.765\n",
      "Ep.:  18 , timesteps:  68, noise_scale (Gt < best(Gt)): 2.0000, Gt 49.5114, \tAvg.Score:  79.111\n",
      "Ep.:  19 , timesteps: 500, noise_scale (Gt >= best(Gt)): 1.0000, Gt 99.3430, \tAvg.Score:  101.263\n",
      "Ep.:  20 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.5000, Gt 99.3430, \tAvg.Score:  121.200\n",
      "Ep.:  21 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.2500, Gt 99.3430, \tAvg.Score:  139.238\n",
      "Ep.:  22 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.1250, Gt 99.3430, \tAvg.Score:  155.636\n",
      "Ep.:  23 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0625, Gt 99.3430, \tAvg.Score:  170.609\n",
      "Ep.:  24 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0312, Gt 99.3430, \tAvg.Score:  184.333\n",
      "Ep.:  25 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0156, Gt 99.3430, \tAvg.Score:  196.960\n",
      "Ep.:  26 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0078, Gt 99.3430, \tAvg.Score:  208.615\n",
      "Ep.:  27 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0039, Gt 99.3430, \tAvg.Score:  219.407\n",
      "Ep.:  28 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0020, Gt 99.3430, \tAvg.Score:  229.429\n",
      "Ep.:  29 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  238.759\n",
      "Ep.:  30 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  247.467\n",
      "Ep.:  31 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  255.613\n",
      "Ep.:  32 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  263.250\n",
      "Ep.:  33 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  270.424\n",
      "Ep.:  34 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  277.176\n",
      "Ep.:  35 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  283.543\n",
      "Ep.:  36 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  289.556\n",
      "Ep.:  37 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  295.243\n",
      "Ep.:  38 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  300.632\n",
      "Ep.:  39 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  305.744\n",
      "Ep.:  40 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  310.600\n",
      "Ep.:  41 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  315.220\n",
      "Ep.:  42 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  319.619\n",
      "Ep.:  43 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  323.814\n",
      "Ep.:  44 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  327.818\n",
      "Ep.:  45 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  331.644\n",
      "Ep.:  46 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  335.304\n",
      "Ep.:  47 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  338.809\n",
      "Ep.:  48 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  342.167\n",
      "Ep.:  49 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  345.388\n",
      "Ep.:  50 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  348.480\n",
      "Ep.:  51 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  351.451\n",
      "Ep.:  52 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  354.308\n",
      "Ep.:  53 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  357.057\n",
      "Ep.:  54 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  359.704\n",
      "Ep.:  55 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  362.255\n",
      "Ep.:  56 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  364.714\n",
      "Ep.:  57 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  367.088\n",
      "Ep.:  58 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  369.379\n",
      "Ep.:  59 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  371.593\n",
      "Ep.:  60 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  373.733\n",
      "Ep.:  61 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  375.803\n",
      "Ep.:  62 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  377.806\n",
      "Ep.:  63 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  379.746\n",
      "Ep.:  64 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  381.625\n",
      "Ep.:  65 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  383.446\n",
      "Ep.:  66 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  385.212\n",
      "Ep.:  67 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  386.925\n",
      "Ep.:  68 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  388.588\n",
      "Ep.:  69 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  390.203\n",
      "Ep.:  70 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  391.771\n",
      "Ep.:  71 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  393.296\n",
      "Ep.:  72 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  394.778\n",
      "Ep.:  73 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  396.219\n",
      "Ep.:  74 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  397.622\n",
      "Ep.:  75 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  398.987\n",
      "Ep.:  76 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  400.316\n",
      "Ep.:  77 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  401.610\n",
      "Ep.:  78 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  402.872\n",
      "Ep.:  79 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  404.101\n",
      "Ep.:  80 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  405.300\n",
      "Ep.:  81 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  406.469\n",
      "Ep.:  82 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  407.610\n",
      "Ep.:  83 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  408.723\n",
      "Ep.:  84 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  409.810\n",
      "Ep.:  85 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  410.871\n",
      "Ep.:  86 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  411.907\n",
      "Ep.:  87 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  412.920\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep.:  88 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  413.909\n",
      "Ep.:  89 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  414.876\n",
      "Ep.:  90 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  415.822\n",
      "Ep.:  91 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  416.747\n",
      "Ep.:  92 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  417.652\n",
      "Ep.:  93 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  418.538\n",
      "Ep.:  94 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  419.404\n",
      "Ep.:  95 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  420.253\n",
      "Ep.:  96 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  421.083\n",
      "Ep.:  97 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  421.897\n",
      "Ep.:  98 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  422.694\n",
      "Ep.:  99 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  423.475\n",
      "Ep.: 100 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  424.240\n",
      "Ep.: 101 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  427.320\n",
      "Ep.: 102 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  431.780\n",
      "Ep.: 103 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  435.960\n",
      "Ep.: 104 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  440.850\n",
      "Ep.: 105 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  445.770\n",
      "Ep.: 106 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  450.450\n",
      "Ep.: 107 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  455.350\n",
      "Ep.: 108 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  460.260\n",
      "Ep.: 109 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  465.160\n",
      "Ep.: 110 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  469.820\n",
      "Ep.: 111 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  474.740\n",
      "Ep.: 112 , timesteps: 500, noise_scale (Gt >= best(Gt)): 0.0010, Gt 99.3430, \tAvg.Score:  479.630\n",
      "Environment solved in 112 episodes!\tAverage Score: 479.63\n"
     ]
    }
   ],
   "source": [
    "env.seed(0)\n",
    "np.random.seed(0)\n",
    "\n",
    "policy = Policy()\n",
    "\n",
    "def hill_climbing(n_episodes=1000, gamma=0.99, print_every=100, noise_scale=1e-2):\n",
    "    \"\"\"Implementation of hill climbing with adaptive noise scaling.\n",
    "        \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        gamma (float): discount rate\n",
    "        print_every (int): how often to print average score (over last 100 episodes)\n",
    "        noise_scale (float): standard deviation of additive noise\n",
    "    \"\"\"\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores = []\n",
    "    arr_noise = []\n",
    "    best_Gt = -np.Inf\n",
    "    best_w = policy.w\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        rewards = []\n",
    "        state = env.reset()\n",
    "        timesteps = 0 ## is the same as len(rewards)\n",
    "        \n",
    "        while True:\n",
    "            action = policy.act(state)\n",
    "            state, reward, done, _ = env.step(action)            \n",
    "            rewards.append(reward)\n",
    "            timesteps += 1  \n",
    "            if done:\n",
    "                break \n",
    "                \n",
    "        total_reward = sum(rewards)        \n",
    "        scores_deque.append(total_reward)\n",
    "        scores.append(total_reward)\n",
    "\n",
    "        discounts = [gamma**i for i in range(len(rewards)+1)]\n",
    "        ## This is the 'cumulative discounted reward' or TD-target G_t\n",
    "        Gt = sum([a*b for a,b in zip(discounts, rewards)])\n",
    "        \n",
    "        if Gt >= best_Gt: # found better weights\n",
    "            ## if Gt > best_R ==> decrease the noise: noise = noise/2  (till 0.001)\n",
    "            best_Gt = Gt\n",
    "            best_w = policy.w\n",
    "            noise_scale = max(1e-3, noise_scale / 2)\n",
    "            print('Ep.: {:3d} , timesteps: {:3d}, noise_scale (Gt >= best(Gt)): {:.4f}, Gt {:.4f}, \\tAvg.Score:  {:.3f}'\\\n",
    "                  .format(i_episode, timesteps, noise_scale, Gt, np.mean(scores_deque)))\n",
    "            policy.w += noise_scale * np.random.rand(*policy.w.shape) \n",
    "        else: # did not find better weights\n",
    "            ## if Gt < best_R ==> increase the noise: noise = 2*noise (till 2)\n",
    "            noise_scale = min(2, noise_scale * 2)\n",
    "            print('Ep.: {:3d} , timesteps: {:3d}, noise_scale (Gt < best(Gt)): {:.4f}, Gt {:.4f}, \\tAvg.Score:  {:.3f}'\\\n",
    "                  .format(i_episode, timesteps, noise_scale, Gt, np.mean(scores_deque)))\n",
    "            policy.w = best_w + noise_scale * np.random.rand(*policy.w.shape)\n",
    "            \n",
    "        arr_noise.append(noise_scale)   \n",
    "\n",
    "        if np.mean(scores_deque)>=threshold:\n",
    "            print('Environment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "            policy.w = best_w\n",
    "            break\n",
    "    \n",
    "    return scores, arr_noise\n",
    "            \n",
    "scores, arr_noise = hill_climbing()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Plot the Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dfZjcZX3v8fd3Zh+S3QSSTTYQ82DABIRaeTBSKh4apVq1PWBbqXpqQY6V46kteLBXi7Y96qm9ak+tqNdprVSswXp4OD4AVS4rRZDaAiWQiDwTKJAlgWzI5nFJdmfme/743b/Z2d3ZzW82c8/M7nxe17XXzPzmN7v3MOH3nfv+3vf3NndHREQEINfsBoiISOtQUBARkTIFBRERKVNQEBGRMgUFEREp62h2A47G0qVLfc2aNc1uhojIrHL//ffvcvf+as/N6qCwZs0aNm3a1OxmiIjMKmb27FTPafhIRETKFBRERKRMQUFERMoUFEREpExBQUREyqIGBTN7xsx+amZbzGxTONZnZreZ2ZPhdnE4bmb2RTPbamYPmtmZMdsmIiKTNaKn8CZ3P93d14fHVwK3u/s64PbwGODtwLrwcynwpQa0TUREKjRjncIFwIZwfyNwJ/CH4fi1ntTyvsfMFpnZcnff0YQ2NtT9z+6mp6uDU5Yfk/k17s437n2OnfsORWyZiLSq8045jtNWLar7740dFBz4gZk58GV3vxo4Lr3Qu/sOM1sWzl0BbKt47UA4Ni4omNmlJD0JVq9eHbn5jfGJWx5mxaL5fPm31h/55GBw/2H++KaHADCL1TIRaVXLjpk3K4PCOe6+PVz4bzOzx6Y5t9qlbdIOQCGwXA2wfv36ObFD0EihxEihVNNrDofz//Jdr+XC9atiNEtE2lDUnIK7bw+3O4HvAGcBL5rZcoBwuzOcPgBUXt1WAttjtq9VFEpOoVRbfCuG8zvy6iaISP1ECwpm1mtmC9P7wFuBh4BbgIvDaRcDN4f7twAXhVlIZwN72yGfAMkFvlCsLSikQSSf06xiEamfmMNHxwHfsWTAuwP4v+7+fTO7D7jRzD4APAdcGM6/FXgHsBUYBi6J2LaWUih6+Zt/VuWeQk49BRGpn2hBwd2fBk6rcvwl4Lwqxx34cKz2tLJiySmUassppOfnFRREpI409tACjiqnoKAgInWkoNACSn40OQUFBRGpHwWFFlAolo4ip6CPUETqR1eUFlAsOaO15hSK6imISP0pKLSAQukoZh9pnYKI1JGCQguY2ToFzT4SkfpTUGgBRa99SqpmH4lIDAoKTVYqOe7UPHyk2UciEoOCQpOlF/eZr1PQRygi9aMrSpOlF3etUxCRVqCg0GRpLqH2nEJyvnIKIlJPCgpNlsaCmnMKWqcgIhEoKDRZ2kMYLTpJTcBstE5BRGJQUGiyyh5CLZ0F5RREJAYFhSarnHVUS15Bs49EJAZdUZqssqdQS15BPQURiUFBockqewqjNUxLLRQ1+0hE6k9BocmOtqegRLOI1JOCQpMVlVMQkRaiK0qTVQaCWlY1pz0FjR6JSD0pKDTZTIePiqUSHTnDTFFBROpHQaHJxk9Jra2noJlHIlJvCgpNVqoMCsUacgpF18wjEak7BYUmU09BRFqJgkKTzTyn4HTk9fGJSH3pqtJk4xevZR8+Uk9BRGJQUGiyYsWU1JnMPhIRqScFhSarXJugnIKINJuCQpOVvHL2UY05BQUFEakzBYUmm2npbPUURCQGBYUmm/Hso6Kr7pGI1F30q4qZ5c1ss5l9Nzw+wczuNbMnzewGM+sKx7vD463h+TWx29YKKoeMaiqdrZ6CiETQiK+alwOPVjz+C+Aqd18HDAEfCMc/AAy5+1rgqnDenHdUtY9UNltE6ixqUDCzlcAvA18Jjw14M/DNcMpG4J3h/gXhMeH586wNqr0VXTkFEWkdsXsKnwf+AEivdkuAPe5eCI8HgBXh/gpgG0B4fm84fxwzu9TMNpnZpsHBwZhtb4hxiWbNPhKRJosWFMzsV4Cd7n5/5eEqp3qG58YOuF/t7uvdfX1/f38dWtpcxeLMFq+ppyAiMXRE/N3nAOeb2TuAecAxJD2HRWbWEXoDK4Ht4fwBYBUwYGYdwLHA7ojtawkzLYhXLDldnfkYTRKRNhatp+DuH3P3le6+BngP8EN3/03gDuBd4bSLgZvD/VvCY8LzP3T37FfJWWp8olk5BRFprmZMdP9D4Aoz20qSM7gmHL8GWBKOXwFc2YS2Ndz4gniqfSQizRVz+KjM3e8E7gz3nwbOqnLOIeDCRrSnlZRmOCW1UFRPQUTqT0tim+xocgqd2k9BROpMV5UmK46bkqqcgog0l4JCkxVKTlf4xl9b6WzlFESk/hqSU5CppeUqim41F8RTT0FE6m3KnoKZLTCz/2VmD5vZXjMbNLN7zOz9DWzfnFcsQT5n5HPGaI1TUlX7SETqbbqewjeA7wC/BPwG0AtcD/yxmZ3k7h9vQPvmvHRqaSlnFGssc6GegojU23Q5hTXu/jV3H3D3zwHnu/uTwCXArzWmeXNfmjDO56zm7Ti1n4KI1Nt0V5WDZvZGADP7z4SSE+5eonqdIpmB9Bt/Rz5XU5VU9RREJIbpho8+BHzFzE4GfkrY98DM+oG/bkDb2kL6jd+9VGNBPM0+EpH6mzIouPuDVF95PAh8MWaj2kn6jd/dai6drZ6CiNTbtFNSzexVwK+SVC8tAE8C17n73ga0rS2k+yI4M8kpKCiISH1NNyX1MuBvScpevx6YTxIc7jazDQ1pXRtIv/F35nKZg0Kp5LhDXolmEamz6XoKHwROd/eimX0OuNXdN5jZl0nKXZ/RkBbOcYVSqTwMlLV0dho8tE5BROrtSF8106DRDSwEcPfngM6YjWonxYopqVlLZ6cJaeUURKTepuspfAW4z8zuAc4F/gLKs4/m/I5ojZLmBsyyl85Op64qpyAi9Tbd7KMvmNk/A6cAn3P3x8LxQZIgIXWQ9hTMsyea1VMQkVimnX3k7g8DDzeoLW2pGNYplNwzl84u5xQUFESkzmY0fcXMvlvvhrSrQsnJ5aipzMVYT0Gzj0SkvmZ6VflgXVvRxtKeQmc+V0NOQT0FEYkj034KZtYHuLsPkdzZEbVVbaRyB7XMPYWicgoiEsd0i9dWm9n1ZjYI3EsyE2lnOLamUQ2c69LS2R05qyGnEGYfaZ2CiNTZdMNHN5Dsp3C8u69z97XAcuAmkn0VpA7STXY68tl3XtPsIxGJZbqgsNTdb3D3YnrA3Yvufj2wJH7T2kMxrGjuqKHMxVhOQYlmEamv6XIK95vZ3wAbgW3h2CrgYmBz7Ia1izSnkLPsw0dFJZpFJJLpgsJFJHsofApYQbKxzgBwC3BN/Ka1h7RKaq6GKanpeXnlFESkzqZb0TwCfCn8SCSFopPP5cjnaihzUVSZCxGJI9OgtJm9r/JW6qfkSU8hn8tlLohXUKJZRCLJmqm8YsKt1EmyotnozFvm0tlFJZpFJJJaryr6alpnaU6hljIX6imISCz6qtlkhWI6JTX7Hs1Flc4WkUgUFJpsrKdQQ+0jlbkQkUiiBQUzm2dm/25mPzGzh83sU+H4CWZ2r5k9aWY3mFlXON4dHm8Nz6+J1bZWUnQnn09yCoVacwqakioidZY1KDwRbh+v4XcfBt7s7qcBpwNvM7OzSXZwu8rd1wFDJGshCLdDoZzGVeG8Oa8yp1ByKGXoLahKqojEkikouPt7Km8zvsbd/UB42Bl+HHgz8M1wfCPwznD/gvCY8Px5Zjbnr3qFkpM3K1/gsySbtZ+CiMSStXT2YmAdMC895u53ZXhdHrgfWAv8NfAUsMfdC+GUAZLV0oTbbeF3F8xsL0mNpV2Z3sksVCo57snFvSOfXOCz5BXUUxCRWI4YFMzst4HLgZXAFuBs4G6Sb/zTCsX0TjezRSQVV0+pdlr6p6Z5rrI9lwKXAqxevfpITWhphYrcQHqBHy2VmE9+2tels4+UaBaRessy/nA58HrgWXd/E3AGMFjLH3H3PcCdJAFlkZmlwWglsD3cHyApuEd4/lhgd5XfdbW7r3f39f39/bU0o+VUlsBOL/DFDNNS1VMQkViyBIVD7n4IkhlC7v4YcPKRXmRm/aGHgJnNB34ReBS4A3hXOO1i4OZw/5bwmPD8D9092xzNWaroYxf3dPiotpyCgoKI1FeWnMJAuLjfBNxmZkOMfbufznJgY8gr5IAb3f27ZvYIcL2ZfZqkBHdacfUa4OtmtpWkh5A5qT1bpb2C3LhE85GnpabrFFTmQkTq7YhBwd1/Ndz9pJndQTKs8/0Mr3uQZKhp4vGngbOqHD8EXHik3zuXVG6rWd6nOcPwUVGls0UkkqxVUheb2WuB/SRj/6+J2qo2UTkM1Bku8Jp9JCLNlGX20Z8C7weeBtKxjXS9gRyFyot7uuYgW05Bs49EJI4sOYXfAF4VNt2ROqpchFZTTiF93dxf2yciDZZl+OghYFHshrSjsaAwNhSUNaeQM8ippyAidZalp/DnwGYze4iknhEA7n5+tFa1iUJlT6HGnIJmHolIDFmCwkaS4nQ/ZSynIHVQrJpTOPJ/4mLJVSFVRKLIEhR2ufsXo7ekDRUqEsa1DB8Viq4ks4hEkSUo3G9mf06y4rhy+OiBaK1qE5U9hTQoZBk+KpZKmo4qIlFkCQrpArSzK45pSmodpAEgl7PycNBoxpyCymaLSAxZVjS/qRENaUfVcgrFrDkF9RREJIIsi9cWARcBayrPd/fL4jWrPRQqVjTXklMYVU5BRCLJMnx0K3APmn1Ud2M9hbEpqVlXNGv2kYjEkCUozHP3K6K3pA1V7SlkzikoKIhI/WXJVn7dzD5oZsvNrC/9id6yNlAaN/tIOQURab4sPYUR4C+BP2Jse0wHTozVqHZR2VNIv/mPZtx5TbOPRCSGLEHhCmCtu++K3Zh2U1nttJYyF+opiEgsWb5uPgwMx25IOypUGT5STkFEmilLT6EIbAm7rlWuaNaU1KNUrDolNUtOQSuaRSSOLEHhpvAjdVa513K+liqpWqcgIpFkWdG8sRENaUdFT8tcUNOU1GLJ6e5UollE6i/LiuZ1JHsqnArMS4+7u2YfHaVxi9fKU1Kz5RR6NPtIRCLIcmX5e+BLQAF4E3At8PWYjWoX1RavjWbKKWj2kYjEkSUozHf32wFz92fd/ZOoQmpdFEMA6MgZuZxhlr2noJyCiMSQJdF8yMxywJNm9rvA88CyuM1qD+WeQkgyd+Zy2WsfKSiISARZegofAXqAy4DXAb8FXByzUe2iFBLNeUsu8PmcZZqSqp6CiMSSZfbRfeHuAeCSuM1pL5U5BUiGkbLOPlJPQURiyDL76B8Zq3mU2gtsAr7s7odiNKwdFItjK5oBOvJWwzoFzT4SkfrLcmV5mqSX8HfhZx/wInBSeCwzNLGnkM/lMhXEK5acTu2nICIRZNqj2d3PrXj8j2Z2l7ufa2YPx2pYOyiG3IDZ2PBRltLZyimISCxZegr9ZrY6fRDuLw0PR6K0qk0USl5OMkMyfKTZRyLSTFl6Ch8FfmxmTwEGnAD8jpn1AiqBcRRKPv4bf0fOMu3RrP0URCSWLLOPbg2lLl5NEhQeq0gufz5m4+a6QnH8LKJ8LluiuVhy7dEsIlFk+rrp7ofd/SfuviXrbCMzW2Vmd5jZo2b2sJldHo73mdltZvZkuF0cjpuZfdHMtprZg2Z25szf1uxQLJXKC9cAOvM5CsopiEgTxRyDKAAfdfdTgLOBD5vZqcCVwO3uvg64PTwGeDuwLvxcSlJvaU4rlCb3FLIMH2mdgojEEi0ouPsOd38g3N8PPAqsAC5gLBexEXhnuH8BcK0n7gEWmdnyWO1rBcVSlZzCEYaP3H3S60RE6qXmoGBmy82su8bXrAHOAO4FjnP3HZAEDsbqKK0AtlW8bCAcm/i7LjWzTWa2aXBwsNbmt5TihNlHWXIKlVt4iojU20x6Cl8HHjOzz2Y52cwWAN8CPuLu+6Y7tcqxSVdId7/a3de7+/r+/v5MDW5VxZKPyyl05HNHLJ09toWnZh+JSP3VfGVx918ETiTZZ2FaZtZJEhC+4e7fDodfTIeFwu3OcHwAWFXx8pXA9lrbl8XNW57n1/7mXzPtXRBTklMY+wg61FMQkSbLFBTM7I1mdkm4vxRY4+7Trma2ZJnuNcCj7v65iqduYazK6sXAzRXHLwqzkM4G9qbDTPX20oERHnhuD8OHizF+fWYTcwP5DDmFtF6ScgoiEkOWgnifANYDJ5P0DrqAfwDOOcJLzyEps/1TM9sSjn0c+Axwo5l9AHgOuDA8dyvwDmArMEzEiqy93XkADowUOLanM9afOaLChJXJWaakps9rnYKIxJBlRfOvkiSJ05lE281s4ZFe5O4/pnqeAOC8Kuc78OEM7Tlqvd3J2z54uNCIPzelYglyVtuU1GJJPQURiSfL8NFIuGA7QChvMau1TlAojfvGr5yCiDRblqBwo5l9mWTdwAeBf2aWl8zu7UqDQnNzChNXJnfkc0cMCpp9JCIxZal99FkzewvJPgonA//T3W+L3rKIyjmFpvcUxq9M7sgZo0fMKainICLxTBsUzCwP/FOYhjqrA0GlBWH4aHikuUFhYk8hn7Py7KKppPstKKcgIjFMOwbh7kVg2MyObVB7GqKnq1VyCuODQmeG/RTUUxCRmLLMPjpEMq30NuBgetDdL4vWqsjSnsLBkVZYpzAWl7OsUyhonYKIRJQlKHwv/MwZ8zpz5Kw1egrjcwo5ChnLXGidgojEkCXRvNHMuoCTwqHH3X00brPiMjN6uzqanmieNPuohimpmn0kIjFkWdG8gaTE9TMki9FWmdnF7n5X3KbF1dvd0QJlLsavaM7njdGMU1KVUxCRGLIMH/0V8FZ3fxzAzE4CrgNeF7NhsfV05znQArOPcjX3FErlc0VE6i3LGERnGhAA3P0JoHkFg+pkQXdH03MKpUk7ryWL15IF5NUppyAiMWUJCpvM7Boz2xB+/g64P3bDYuvtav7w0cScQme4P3EG0nc2D/DRG38y7jnlFEQkhixXlv8OPAxcBlwOPAJ8KGajGqG3O9/0RPPE2UfphjsTh5C+/9AL/OOD25OtOIvKKYhIPFlyCh3AF9I9EcIq55q242xFvd0dHGyBnEJ+wiY76fFK23a/zEihxPBIsaKnoKAgIvWXpadwOzC/4vF8kqJ4s1pvd0fTC+JVW6cAjFur4O5s2z0MwO6DI5p9JCJRZQkK89z9QPog3O+J16TG6O3KNz3RXCiWJlRJndxT2DM8yv7QzqHhkfLsI/UURCSGLEHhoJmdmT4ws9cBL8drUmP0dnfw8mjxiFNAYyo5kwriwficwrah4fL98T0FJZpFpP6y5BQ+Avw/M9seHi8H3h2vSY1RWSl14bzmzLCdtB1nuNCPVgwfPbd7LCgMDY+M1T7SlFQRiSBLmYv7zOzVJHspGPDYbC9zAZWVUotNCwoTq6RW7SnsHuuU7T44yvzOZC8I5RREJIYpxyDM7PVmdjxACAJnAp8G/srM+hrUvmhaYaOdwsREc5WcwnO7h1nU00k+ZwwdHNF+CiIS1XQD018GRgDM7FzgM8C1wF7g6vhNi6vZG+2USo47E8pcpLOPKnsKw7xySS+LezrZPTyi/RREJKrpho/y7r473H83cLW7fwv4lpltid+0uNLho2b1FIo++eKeL69TGMspbBsa5mdXHMvw4ULoKWidgojEM11PIW9madA4D/hhxXNZEtQtrbzRTpPWKhSrlKvomJBTKJac54deZnVfD4t7u9h9sLKnoNlHIlJ/011ZrgN+ZGY3k0xB/RcAM1tLMoQ0q6U5hWYNH1UbBpqYU9ix92UKJWd1Xw99PV3jpqSqpyAiMUz5jd/d/8zMbieZgvoDHyvdmQN+rxGNi6m3u8nDR1W21ZyYU0ino64KPYWhZ8empCqnICIxTDsM5O73VDn2RLzmNE5vefioWT2FybOIJuYU0vIWq/t66OvtZGh4lNFiCbPxCWoRkXpp24HpnjDfv/k5hcnDR+lz23a/TD5nLD92Hot7uiiWnKHhEfUSRCSatg0KuZzR08T6R9VmH5WrpFYMH71i0Tw68jn6ersAGNx/WPkEEYmmbYMCNLd8dmG6nEJpLCisWpzUHlycBoUDhzXzSESiaeury4Imls+utq3mWJmLJKcwMDTM6r4kKPT1qKcgIvG1dVBo5vBR2hvIWUVBvBAgRovOwcMFdh0YYVUaFCqGj5RTEJFYogUFM/uqme00s4cqjvWZ2W1m9mS4XRyOm5l90cy2mtmDlaW6Y+rt7mjelNQqi9AqC+INDCWF8CYGhcOF0rjehYhIPcXsKXwNeNuEY1cCt7v7OpId3a4Mx98OrAs/lwJfitiust6uPMMjLTT7qCKncM/TLwGUh496uvJ0deTGnSciUm/Rri7ufhewe8LhC4CN4f5G4J0Vx6/1xD3AIjNbHqttqWRLzmb3FCZPSb37qZf4s+89ytkn9vGzK44FwMzKeQXlFEQklkZ/5TzO3XcAhNtl4fgKYFvFeQPh2CRmdqmZbTKzTYODg0fVmAVNHD4qL17LT56S+q0HBli5eD5/+77XjQsA6Qwk5RREJJZWGYeodpWruk+mu1/t7uvdfX1/f/9R/dGero7mDx/Z5NlHi3s6+er7X8+i0DNI9fV2jjtPRKTeGl3t9EUzW+7uO8Lw0M5wfABYVXHeSmD7pFfX2YLuPAdHCrg7Zo290FYriLe4p4v3v2EN55/+CtYs7Z30msUaPhKRyBrdU7gFuDjcvxi4ueL4RWEW0tnA3nSYKabe7g7caUpvoVqiOZczPnn+z3Dm6sVVX5POQNLsIxGJJVpPwcyuAzYAS81sAPgEye5tN5rZB4DngAvD6bcC7wC2AsPAJbHaVaknLYo3UigXyGuUaovXjmSsp9Aqo34iMtdEuxK6+3uneOq8Kuc68OFYbZnKgu6KongLG/u3q22ycyR9SjSLSGRt/ZWzt2t8+exvPzDAf+w62JC/PZO9ltPZR8opiEgs7R0UKvZUeOnAYa648Sd8/e5nG/K30/pGuRoS3EvUUxCRyBQUSHIK9z2TrLPbuf9QQ/524ahyCgoKIhJHo6ektpTKnMIDzw0BsHP/4Yb87ZnstaycgojE1tY9hZ6KnMK9Tyc9hV0NDgq1XOAX9aSL19r6YxORiNr66pIOH23fe4hHX9hHPmcN6ykUZtBTmNeZp7crr56CiETT3kGhKxk+uuuJQdzhnLVLOXC4wHADdmObyfARQN+CLjo72vpjE5GI2vrq0pHP0d2R4ycDe+jMG2899Tgg2cgmtpn0FAD+9ILX8N/OPTFGk0RE2jsoQFIp1R1OW7movHdBI4aQisVkSmqteyNsOHkZrwnltEVE6q3tg0KaVzjrhD76F3YDjekpFEMNWE0vFZFW0vZBoSfkFc46oY9lISjs3Bd/rUK6eE1JYxFpJW29TgGS4aOcwfo1ffR0JjN7Bg+0bk5BRCSmtg8KKxfPpyNvLAjDSEsXdLNzXyNyCgoKItJ62j4ofObXX1ueHgqw7JjuhiSaC1V2XhMRaba2DwrzOvPjHvcv6GbH3vg5hZI7OUs21hERaRVtn2ieqJE9hVqno4qIxKar0gT9C+ex++DhcUNKMRRLjmKCiLQaXZYm6F/YTcnhpcgzkHYdOMzCeZ1R/4aISK0UFCYor1WIPIS05bk9nLZyUdS/ISJSKwWFCeq1qnnn/kNT9jaGDo7w9K6DnPlKBQURaS0KChOM9RQmz0B6cGAPDw7syfR7/uvX7uPy67dUfW5L+B1nrFo8w1aKiMTR9lNSJ5qup/CRG7ZQLDl3/v4GbJr1BTv2vsxDz++jM28MjxTKm/mkNj87RM7gtStV2E5EWot6ChN0d+Q5dn7npJzCtt3DPD14kGdfGubJnQem/R0/enwQgNGic98zQ5Oe37xtD68+/phyMT4RkVahoFDFsoWTS13c+fjO8v3bHnlx2tf/6IlBli3spiuf49+27hr3XKnkbHluD2esVj5BRFqPgkIV/Qu7JxXFu/PxQVb1zee0VYv4wcMvTPna0WKJHz+5i/NOWcYZqxfx4wlBYevgAfYfLnDmauUTRKT1KChUsWxh97hE86HRIv/21EtsOGkZbz31OH4ysJcXpiiF8cCzQ+w/XOAXTurnnLVLeWTHPoYOjpSf3/xcMpyknoKItCIFhSr6F3YzuP8w7smq5k3PDPHyaJENJ/eXt+y87dHqQ0h3PjFIR844Z+1Szlm7BHe4++mXys8/8OweFvV0csLS3vhvRESkRgoKVSxbOI9DoyX2Hy4AST6hK5/j51+1hLXLFrBmSc+UeYU7Hx/kda9czMJ5nbx25SIWdHfwrxVDSJu3DXHGqkXTzl4SEWkWBYUqlh2T7sCW5BXufGKQs07oo6erAzPjrT9zPHc/tYv9h0bHve7FfYd4dMc+Npy8DIDOfI6fO6GPf3sq6SnsOzTKkzsPcIbyCSLSojQnsorjj5kHwKe/9wi/fuZKtu48wHtev6r8/FtOPY6r73qaP7npIc4+cQnHHzuPHXsPlZPKG07uL5/7hrVLuf2xndy85Xm+s/l53FGSWURaloJCFWed0Mfl561j493PcGdYc1B5oT9z9WLe8Kol3PrTF7hpy/by8a6OHOee1M+rj19YPnbO2iUAXH79Fvp6u7jiLSfxhlctacwbERGpkaXJ1FZgZm8DvgDkga+4+2emO3/9+vW+adOmaO15eaTItzcPsGv/CJedt3ZSHqBYcrbveZkX9h3i+GPm8YpF8ydtr+nuXPXPT/KKY+fxzjNWTNrUR0Sk0czsfndfX/W5VgkKZpYHngDeAgwA9wHvdfdHpnpN7KAgIjIXTRcUWinRfBaw1d2fdvcR4Hrggia3SUSkrbRSUFgBbKt4PBCOjWNml5rZJjPbNDg42LDGiYi0g1YKCtUm7k8a23L3q919vbuv7+/vr/ISERGZqVYKCgPAqorHK4HtU5wrIiIRtFJQuA9YZ2YnmFkX8B7glia3SUSkrbTMOgV3L5jZ7wL/RDIl9avu/nCTmyUi0lZaJigAuPutwK3NboeISLtqpeEjERFpsht+6JsAAAZPSURBVJZZvDYTZjYIPFvjy5YCu4541uwzV98X6L3NVnpvreuV7l51+uasDgozYWabplrJN5vN1fcFem+zld7b7KThIxERKVNQEBGRsnYMClc3uwGRzNX3BXpvs5Xe2yzUdjkFERGZWjv2FEREZAoKCiIiUtY2QcHM3mZmj5vZVjO7stntORpmtsrM7jCzR83sYTO7PBzvM7PbzOzJcDsrN4M2s7yZbTaz74bHJ5jZveF93RBqY81KZrbIzL5pZo+Fz+/n58LnZmb/I/xbfMjMrjOzebP1czOzr5rZTjN7qOJY1c/IEl8M15UHzezM5rW8PtoiKIRd3f4aeDtwKvBeMzu1ua06KgXgo+5+CnA28OHwfq4Ebnf3dcDt4fFsdDnwaMXjvwCuCu9rCPhAU1pVH18Avu/urwZOI3mfs/pzM7MVwGXAend/DUntsvcwez+3rwFvm3Bsqs/o7cC68HMp8KUGtTGatggKzLFd3dx9h7s/EO7vJ7mwrCB5TxvDaRuBdzanhTNnZiuBXwa+Eh4b8Gbgm+GUWfm+AMzsGOBc4BoAdx9x9z3Mgc+NpI7afDPrAHqAHczSz83d7wJ2Tzg81Wd0AXCtJ+4BFpnZ8sa0NI52CQqZdnWbjcxsDXAGcC9wnLvvgCRwAMua17IZ+zzwB0ApPF4C7HH3Qng8mz+7E4FB4O/D8NhXzKyXWf65ufvzwGeB50iCwV7gfubO5wZTf0Zz7trSLkEh065us42ZLQC+BXzE3fc1uz1Hy8x+Bdjp7vdXHq5y6mz97DqAM4EvufsZwEFm2VBRNWF8/QLgBOAVQC/JsMpEs/Vzm85c+vcJtE9QmHO7uplZJ0lA+Ia7fzscfjHtuobbnc1q3wydA5xvZs+QDPG9maTnsCgMS8Ds/uwGgAF3vzc8/iZJkJjtn9svAv/h7oPuPgp8G3gDc+dzg6k/ozl3bWmXoDCndnUL4+zXAI+6++cqnroFuDjcvxi4udFtOxru/jF3X+nua0g+ox+6+28CdwDvCqfNuveVcvcXgG1mdnI4dB7wCLP8cyMZNjrbzHrCv830fc2Jzy2Y6jO6BbgozEI6G9ibDjPNVm2zotnM3kHyrTPd1e3PmtykGTOzNwL/AvyUsbH3j5PkFW4EVpP8j3qhu09MmM0KZrYB+H13/xUzO5Gk59AHbAbe5+6Hm9m+mTKz00mS6F3A08AlJF/OZvXnZmafAt5NMjNuM/DbJGPrs+5zM7PrgA0k5bFfBD4B3ESVzygEwf9DMltpGLjE3Tc1o9310jZBQUREjqxdho9ERCQDBQURESlTUBARkTIFBRERKVNQEBGRMgUFaUtmVjSzLRU/064sNrMPmdlFdfi7z5jZ0hm87pfM7JNmttjMbj3adohMpePIp4jMSS+7++lZT3b3v43ZmAz+E8lisHOBf21yW2QOU1AQqRBKbNwAvCkc+i/uvtXMPgkccPfPmtllwIdIFmo94u7vMbM+4KskRe+GgUvd/UEzWwJcB/QD/05FrRwzex9JyekukoWHv+PuxQnteTfwsfB7LwCOA/aZ2c+5+/kx/htIe9PwkbSr+ROGj95d8dw+dz+LZKXq56u89krgDHd/LUlwAPgUsDkc+zhwbTj+CeDHoQDeLSQrYjGzU0hWAJ8TeixF4Dcn/iF3v4GkPtJD7v6zwEPhbysgSBTqKUi7mm746LqK26uqPP8g8A0zu4mk/AHAG4FfB3D3H5rZEjM7lmS459fC8e+Z2VA4/zzgdcB9SaUE5jN1Ibx1wFPhfk/YQ0MkCgUFkcl8ivupXya52J8P/ImZ/QzTl1Cu9jsM2OjuH5uuIWa2iaQGT4eZPQIsN7MtwO+5+79M/zZEaqfhI5HJ3l1xe3flE2aWA1a5+x0kmwEtAhYAdxGGf0Ixv11hj4vK428H0v2XbwfeZWbLwnN9ZvbKiQ1x9/XA90jyCf8b+CN3P10BQWJRT0Ha1fzwjTv1fXdPp6V2m9m9JF+a3jvhdXngH8LQkJHsQbwnJKL/3sweJEk0p2WWPwVcZ2YPAD8iqbCJuz9iZn8M/CAEmlHgw8CzVdp6JklC+neAz1V5XqRuVCVVpEKYfbTe3Xc1uy0izaDhIxERKVNPQUREytRTEBGRMgUFEREpU1AQEZEyBQURESlTUBARkbL/D1k/mm8o3smPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score, gamma=0.99')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxkZX3v8c+3q3u6Z2NmmBlgmIWB64iirHYAgxoxBsEFkmgiGJUQvRMTl2i8JhoTRYz3ahbjzXUL0RE0ETWuY4Ig7kbEMCOL7EyQZZiB2dfeq3/3j3Oqp2iqq05V1+nq6vm+X696dZ2t6qkpOL96nt+zKCIwMzMbr6PVBTAzs+nJAcLMzCpygDAzs4ocIMzMrCIHCDMzq6iz1QVopiVLlsTq1atbXQwzs7axcePGHRGxtNKxGRUgVq9ezYYNG1pdDDOztiHpoYmOuYnJzMwqcoAwM7OKHCDMzKwiBwgzM6vIAcLMzCrKLUBIWinp+5LulnSnpD+pcI4k/aOkTZJul3RG2bFLJd2fPi7Nq5xmZlZZnt1cR4C3R8TPJc0HNkq6ISLuKjvnAmBN+jgL+ARwlqQjgfcCvUCk166PiN05ltfMzMrkFiAiYiuwNX2+X9LdwHKgPEBcBHw2kjnHb5K0UNIy4PnADRGxC0DSDcD5wDV5lTdPA8NFPvOTB+kfGpnwnEJHB5ectZKj5veM7fvljoN87ZZHocqU7AvnzOKyc1YjqallNjObkoFyklYDpwM/G3doOfBI2fbmdN9E+yu99lpgLcCqVauaUt5mu+mBnXzounsAqHQfL93/53YXeP1zTxjbf/WND3LVjQ9WvKb8uuc9dSlPOWpeM4tsZpZ/gJA0D/gK8NaI2Df+cIVLosr+J++MuBK4EqC3t3darn7UN1QE4Pq3Po8Tj5n/pOPDxVHWvPtbDAwXn7B/YLjI0Ud087O/eGHF1/32nY+x9nMbn3SdmVkz5NqLSVIXSXD414j4aoVTNgMry7ZXAFuq7G9L/WmAmN1VqHi8s0NIMDQy+oT9QyOjzOqc+CsqHRsqjk54jplZo/LsxSTg08DdEfHhCU5bD7w27c10NrA3zV1cD5wnaZGkRcB56b621J/+wu+ZVfmfWxKzCh0MjrvRDxZHmVXIECBGHCDMrPnybGI6B3gN8AtJt6b7/gJYBRARnwSuBV4MbAL6gMvSY7skvR+4Ob3uilLCuh2VmoAmqkFAcrOvXIOY+JpuBwgzy1GevZj+k8q5hPJzAnjjBMfWAetyKNqUKzUx9VQJEN0TBogqNYhCYew8M7Nm80jqKdA/XKSrILqqNRcVOhgcd6MfHCnSnaGJafx1ZmbN4AAxBQaGR6vWHqBaE1OWJLV7MZlZ8zlATIH+4WJjAaKYMUC4BmFmOXCAmAIDw8WqCWpIA0SxQg2iRrNU6Twzs2ZzgJgC/UO1A0R3Z6FiE1N3l3MQZtYaDhBToH+4SM+sGjWIwgQ5iCo1iG4PlDOzHDlATIH+4SKzq9QEIKkNjB8oVzMH4SYmM8uRA8QUyJyDeFI31+oBoqNDdBXkAGFmuXCAmAIDmXsxPbG7aq1urlC5acrMrBkcIKZAf4YaRHfhib2YIoLBkdGqA+Wgcu8nM7NmcICYAv1Do7WT1OOamIaLMba/1nWDww4QZtZ8DhBTIGsOory7aqlWkCVAuAZhZnlwgMhZRGRqYhqfSyg9r9bNtdJ1ZmbN4gCRs+FiUBwNZtfZxDQWIKpM91067oFyZpYHB4icjS0WlGEk9choMDqa5B5KAaLbTUxm1iIOEDkbHAsQtW/0cCj3UJqhtVYOorvw5O6xZmbNkOeSo+skbZN0xwTH3yHp1vRxh6SipCPTYw9K+kV6bENeZZwK/RlWk4Mnz6s0OJItSd3d5RyEmeUjzxrEVcD5Ex2MiL+NiNMi4jTgXcAPxy0rem56vDfHMuau3gBRutkPZQwQswpuYjKzfOQWICLiR0DWdaQvAa7JqyytNLbcaI0kdWlAXOlmX6pBZBoo5xqEmeWg5TkISXNIahpfKdsdwLclbZS0tsb1ayVtkLRh+/bteRa1IbnXIBwgzCwnLQ8QwMuAn4xrXjonIs4ALgDeKOl5E10cEVdGRG9E9C5dujTvstZtYAqamNzN1czyMB0CxMWMa16KiC3p323A14AzW1CupugfSm7eNSfrK5SS1ElAqWsktQOEmeWgpQFC0gLg14BvlO2bK2l+6TlwHlCxJ1Q7mHQNwjkIM2uRzrxeWNI1wPOBJZI2A+8FugAi4pPpab8FfDsiDpZdejTwNUml8n0+Iq7Lq5x5GxsoNyvjOIgGchDjFxoyM2uG3AJERFyS4ZyrSLrDlu97ADg1n1JNvaw1iNKI6dLNvvS3u8ZUG93pXEwRQRpUzcyaYjrkIGa0sW6uOfZigkPTg5uZNYsDRM76h4t0FURXjVxC97gAUUpWZ5mLCfBgOTNrOgeInPVnWG4UYFYhOafeJHWpCcqJajNrNgeInGVZjxoqTNY3Mkpnh+joqJ5XGN80ZWbWLFWT1JLOBV4OrARGgPuBT0XEpiko24wwMDxaM0ENlXMQtfIPcKiG4QBhZs024R1I0geB1wI3AcPAA8B/A/8m6Xempnjtr3+o9mpyUCFAFDMGiM4nDrAzM2uWajWIl0TEyQCSvkAy2+o7JH0Z+DHwb1NRwHbXP1ysOVEfVBhJPTJaM/8AT54m3MysWardgUZL6zMAxwIFgIjYDbjDfUbJetS1b/RdheSftO4mJvdiMrOcVKtB/G/gFkn3Ak8D/ghA0lLgtiko24wwMFxk8dxZNc+T9IRR0YMZm5i6nYMws5xMGCAi4ouSbgBOADZFxJ50/3bgVVNUvrbXP1SkZ2HtJiZIxjyU1yBqjaIG92Iys/xU/YlaNgX3uZJeJulpU1CmGSVpYqo/QAzW28TkAGFmTTZhDULSrwF/D+wBngX8BFgkaRh4TUQ8MjVFbG8DGZPUkC4fOlaDKNZcTQ6cgzCz/FS7A30EuCAiXgicAQxHxDnAB4BPT0XhZoKs4yAgnbq76HEQZjY9VLsDFdJ8A8DDwHEAEXEDsDzvgs0EEVFXE1P52g5Zx0F0d3mqDTPLR7VeTBskfRr4LnAR8AMYW0M62x3vMDdcDIqjweysTUzjktSZxkGUxk+4icnMmqzaHegPgY3ArwLfAd6R7g/gRTmXa0YYWywoaw2i0EATU2mg3LBHUptZc014B4qI4Yj4eES8KSL+OSKK6f7+iHio1gtLWidpm6SKy4VKer6kvZJuTR/vKTt2vqR7JW2S9M5GPth0MDAWILLNiTirs4PB4foCRLeT1GaWk4Zmc5V0eYbTrgLOr3HOjyPitPRxRfraBeBjwAXAScAlkk5qpJytVlosKHsOojDWVJR5LiYnqc0sJ41O972x1gkR8SNgV63zKjiTZGDeAxExBHyBJAfSdvozLjdaMn4cRK3FggA6OkRnhxwgzKzpGgoQEfHNJr3/syXdJulbkp6R7lsOlI+x2EyVXlOS1kraIGnD9u3bJzqtJcaamOpKUifXZB0od+g6Bwgza65q0313SvpDSddJur3sRv4GSV1NeO+fA8dFxKnA/wO+XnrrCudOuOByRFwZEb0R0bt06dImFKt56q5BpEnqiEim2sjQiwmeOH7CzKxZqnVz/RzJKOrLSX7FA6wALgX+BXjlZN44IvaVPb9W0sclLUnfa2XZqSuALZN5r1YZqDNAlGoCw8UY2850XcE1CDNrvmoB4oyIOHHcvs3ATZLum+wbSzoGeDwiQtKZJLWZnSRBaY2k44FHgYtp08kB+4eSm3bmbq5pgCjVBtzEZGatVC1A7E5XjvtKRIwCSOoAfgfYXeuFJV0DPB9YImkz8F6gCyAiPgm8AvgjSSNAP3BxRAQwIulNwPUkA/LWRcSdDX6+lqq3ialUEyjd7LMMlAOeME24mVmzVAsQFwMfAj4uqRQQFgLfT49VFRGX1Dj+UeCjExy7Fri21ntMd2MD5WbVl0sYCxAZpvsGNzGZWT6qrQfxIGmeQdJiQBGxY4rKNSMM1D0OooPhYozlLrI2MXV3FbzkqJk1XdZurksjYofXg6jPQL1TbaQB4cDgyBO2a+kuHOoea2bWLFkDxOfH/bUM+oeLdBVEV9ZcQnre/oGRJ2zXvM5JajPLQb0D5SqNUbAJ9A8XM9ce4NDU3aUaRHcdczh5HISZNVujU21YBgP1BohCqYlp+AnbtThJbWZ5cIDIUf9Q9sWCoCwHMVBfDsJNTGaWh3oDxIRTXtiT1bOaHBwKCPvrTFI7QJhZHrIGCI37axn0D49mnqgPDiWlG6pBOAdhZk2WNUA8d9xfy2BguMjsjIlmqNDNtY4chMdBmFmzZboDRcSB8r+WzUCDTUz11iC63cRkZjmoeQeSdLakmyUdkDQkqShpX63rLElS19OLqRQQ9jXQxDQ4kkwTbmbWLFnuQB8FLgHuB2YDrydZv8FqqDtJPTZQrtTNNftKdMDYNOFmZs2QtYlpE1CIiGJEfAY4N99izQwDw8W6ktTdDU61UTrPiWoza6Zqs7mW9EmaBdwq6W+ArcDcfIs1M9Q7DqK784kjqetZMAhI8hDddRbSzGwCWe5AryFZl+FNwEGS1d5enmehZoKIaHgcxIGBETo7RKEjW6/i0rTgTlSbWTPVrEFExEPp037gfVlfWNI64KXAtoh4ZoXjvwf8ebp5APijiLgtPfYgsB8oAiMR0Zv1faeLoeIoowE9DXRz3T84krn2UH6dA4SZNVOWXkwvlXSLpF2S9knan7EX01XA+VWO/xL4tYg4BXg/cOW44+dGxGntGBwgaV4CmD0rSyteovxG31CAKHrKbzNrnix3r48Avw38IuroRxkRP5K0usrxG8s2bwJWZH3tdtCXBoi5DYykHv8863UeLGdmzZTlLvQIcEc9waEBrwO+VbYdwLclbZS0ttqFktZK2iBpw/bt23MsYn36xmoQ2QNEV+FQzqGeGkS3m5jMLAdZahB/Blwr6YfAYGlnRHy4GQWQdC5JgHhO2e5zImKLpKOAGyTdExE/qnR9RFxJ2jzV29s7bQYClJqY5tTRxCRpbOI95yDMrNWy3IU+APQBPcD8ssekSToF+BRwUUTsLO2PiC3p323A14Azm/F+U6lvKOmqOqeOGgQcWgOiriamTjcxmVnzZfl5e2REnNfsN5a0Cvgq8JqIuK9s/1ygIyL2p8/PA65o9vvnrZEmJkhv9oOHmo0yXVNwDcLMmi9LgPiOpPMi4tv1vLCka4DnA0skbQbeC3QBRMQngfcAi4GPS4JD3VmPBr6W7usEPh8R19Xz3tNB31gTUwMBgjpzEF0eSW1mzZclQLwR+DNJg8AwyZoQERFHVLsoIi6pcfz1JPM6jd//AHBqhnJNa2NNTF3ZcxBwqOZQGlWdhWsQZpaHLAPlmpJvONz0D0+iiYn6ahBOUptZHjL9vE2TyavLz4+Ir+ZUphlh0k1MjSSp3cRkZk1UM0CkU2acAtwJlO5AQZJgtgmMJanrmIsJDgWGunIQBc/FZGbNl6UGcXZEnJR7SWaY/qERZncV6Mg44V6Jm5jMbLrIchf6qSQHiDr1DRXrbl6CQzOzOkCYWatlqUFcTRIkHiMZSV3qxXRKriVrc/1DxboT1FDWxFRHDqKQTg3uyfrMrJmyBIh1JGtC/IJDOQirodEaxKFurtkDBCQBZXDYX4+ZNU+WAPFwRKzPvSQzTN9wsa6pvksayUGUzvdAOTNrpix3sHskfR74Jk+crM+9mKroGxxhTp09mKCxJiZgbJI/M7NmyRIgZpMEhvL5mNzNtYa+oSLLFnTVfV1p2ozuOlaig6RJygHCzJopy0jqy6aiIDNN//DUJakhqUF4oJyZNVOWgXI9JOs1PINkym8AIuIPcixX2+sbGmmwm2spB1H/ADvXIMysmbL8TP0ccAzwIuCHJEuD7s+zUDNB0otp6pLUbmIys2bLchd6SkT8FXAwIq4GXgKcnG+x2l9/wwPlJtGLyQHCzJooy11oOP27R9IzgQUkE/fZBIZGRhkZjcYCxGR6MTkHYWZNlKUN5EpJi4C/AtYD80gW+7EJ9I+tJld/E9NkBsrt6x+p+/3MzCZS8y4UEZ+KiN0R8cOIOCEijkpXhKtJ0jpJ2yTdMcFxSfpHSZsk3S7pjLJjl0q6P31cmv0jtV7fcGPrUcPkmpgGRzzVhpk1T5ZeTH9aYfdeYGNE3Frj8quAjwKfneD4BcCa9HEW8AngLElHkixR2ksy5mKjpPURsbtWeaeDRteCgMkEiIJzEGbWVFnuQr3AG4Dl6WMtyVrT/yzpz6pdGBE/AnZVOeUi4LORuAlYKGkZSY+pGyJiVxoUbgDOz1DWaaFvsLG1IABmpWs71J2DcDdXM2uyLHehxcAZEfH2iHg7ScBYCjwP+P1Jvv9y4JGy7c0cCkSV9j+JpLWSNkjasH379kkWpznG1qNuIAdxyooFnHX8kRy/dG5d182e1TG2zKmZWTNkCRCrgKGy7WHguIjop2xupgZVWk0nqux/8s6IKyOiNyJ6ly5dOsniNEdfg+tRA6w8cg5f/MNnc0RPfdN0zOvu4sDgCBEV/5nMzOqW5Sfu54GbJH0j3X4ZcI2kucBdk3z/zcDKsu0VwJZ0//PH7f/BJN9rypR6Mc3trj9ANGp+TyfDxWBwZJSeBpq2zMzGy9KL6f3A/wT2kCSn3xARV0TEwYj4vUm+/3rgtWlvprOBvRGxFbgeOE/SorSL7XnpvrYwlqTuqr+JqVHze5L3OjDorq5m1hyZ7mARsRHYWO+LS7qGpCawRNJmkp5JXelrfhK4FngxsAnoAy5Lj+2S9H7g5vSlroiIasnuaaU/zUE00sTUqHndaYAYGGHJvO4pe18zm7ly/YkbEZfUOB7AGyc4to5kNbu2M5luro0aCxCuQZhZk9TXl9IyKQWIRrq5Nmpe2sS0b2C4xplmZtk4QOSgf7hIT1cHHR2VOmPlo9Tr6cCAaxBm1hx1BwhJ35H0LUkvzaNAM0GyFsTUJajBTUxm1nyN3MVeCywDzm5yWWaMvqHilDYvwaEmJgcIM2uWTAFC0mxgVUTcGxFbSMYq1N2r6XDRN9jYWhCTUapB7HcTk5k1Sc0mJkkvA24Frku3T5O0Pu+CtbO+4akPED1dBWYVOhwgzKxpsuQgLgfOJBkoRzqD6+r8itT++luQg4CkmenAoHsxmVlzZAkQIxGxN/eSzCB9DS43Olnzujvdi8nMmibLz9w7JL0KKEhaA7wFuDHfYrW3/qHilI6iLpnX3ekktZk1TZYaxJuBZ5DM3HoNsA94a56Fanctq0H0dDoHYWZNU7MGERF9wLvTh2XQinEQAEf0dLJlz8CUv6+ZzUwT3sUkfZMJ1mAAiIgLcynRDNA/7CYmM2t/1X7m/t2UlWIGGS6OMlwM5rRgTYakF5MDhJk1x4QBIiJ+OJUFmSnGJuprSQ2iy72YzKxpajaUpz2X/g9wEtBT2h8RJ+RYrrY1mfWoJ2t+TydDxVEGR4p0d3pVOTObnCy9mD4DfAIYAc4FPgt8Ls9CtbO+Fiw3WlJaVc49mcysGbIEiNkR8V1AEfFQRFwOvCDLi0s6X9K9kjZJemeF4/8g6db0cZ+kPWXHimXH2mZqj/4WrAVRUr6qnJnZZGVpBxmQ1AHcL+lNwKPAUbUuklQAPgb8BrAZuFnS+oi4q3RORLyt7Pw3A6eXvUR/RJyW7WNMH4dWk2vBVBue8tvMmihLDeKtwBySEdTPAl4NXJrhujOBTRHxQEQMAV8ALqpy/iUkA/HaWl8L1qMumecmJjNroiwD5W5Onx4ALqvjtZcDj5RtbwbOqnSipOOA44Hvle3ukbSBJPfxwYj4+gTXrgXWAqxataqO4uWjvwXrUZfM705XlXMNwsyaIMt03zdIWli2vUjS9Rleu9J6mxMNvLsY+HJEFMv2rYqIXuBVwEck/Y9KF0bElRHRGxG9S5cuzVCsfPW1MkCM1SA8o6uZTV6WJqYlETGWPI6I3WTIQZDUGFaWba8gWWiokosZ17yULkxERDwA/IAn5iemrb7hFo6D8KpyZtZEWQLEqKSxtpu0OWjCKTjK3AyskXS8pFkkQeBJvZEknQgsAn5atm+RpO70+RLgHOCu8ddOR/0tHAfhVeXMrJmy3MXeDfynpNLI6ueRtvlXExEjaa+n64ECsC4i7pR0BbAhIkrB4hLgCxFRHnSeDvyTpFGSIPbB8t5P01lfC7u5dnd20FWQaxBm1hRZktTXSToDODvd9baI2JHlxSPiWuDacfveM2778grX3QicnOU9ppv+oSLdnR0UOiqlYPIlifk9Xc5BmFlTTNjEJOk4SQsA0oBwkGRMw2vTJiOr4ODQCHO7p755qcSryplZs1TLQXwJmAsg6TTg34CHgVOBj+dftPbUN1RsSfNSiaf8NrNmqfZTd3apJxHJ4Lh1EfH36ajqW/MvWnvqb9FqciVeVc7MmqVaDaK8Ef0FwHcBImI01xK1uVYtN1oy3zUIM2uSajWI70n6ErCVpBvq9wAkLQOGpqBsbal/qDWryZXM7+nk/m0OEGY2edVqEG8Fvgo8CDwnIkpdY47B61NPaFffEIvmtC6H71XlzKxZqq0oFyQT7I3ff0uuJWpzOw8M8uwTFrfs/b2qnJk1S5aR1JbRSHGU3X3DLJ7XuhpE+apyZmaT4QDRRLv6ktTM4nndLSuDFw0ys2bJFCAkzU7nTLIqduxPAsSSua2tQYDnYzKzycsy3ffLSMY9XJdun9ZOS4BOpZ0HB4FpUoNwotrMJilLDeJyktXh9gBExK3A6vyK1L52HkhrEC3MQXhVOTNrliwBYiQi9uZekhlgx4HW1yC8qpyZNUuWWeXukPQqoCBpDcna1DfmW6z2tPPgEF0FcURPCyfrG1s0yDO6mtnkZKlBvBl4BjBIsurbPpJBdDbOzgODLJ7bjTT1U32XOEltZs1SM0BERF9EvDsifgU4C/hQRAxkeXFJ50u6V9ImSe+scPz3JW2XdGv6eH3ZsUsl3Z8+Lq3nQ7XKjgNDLR0DAV5VzsyaJ0svps9LOkLSXOBO4F5J78hwXQH4GHABcBJwiaSTKpz6xYg4LX18Kr32SOC9JAHpTOC9khZl/lQtsvPAYEvzD+BV5cysebI0MZ0UEfuA3yRZHW4V8JoM150JbIqIByJiiGTajosylutFwA0RsSsidgM3AOdnvLZldhwYamkPJkhWlfOiQWbWDFkCRJekLpIA8Y100r6ocQ3AcuCRsu3N6b7xXi7pdklflrSyzmunjYhg58FBlrS4BgEwv6eLvf1OUpvZ5GQJEP9EMqPrXOBHko4jSVTXUilTOz6wfBNYHRGnAN8Brq7j2uREaa2kDZI2bN++PUOx8tE3VGRgeJTFLRxFXXL0Ed08vi9TmsjMbEJZktT/GBHLI+LFkXgIODfDa28GVpZtrwC2lJ8QETsjYjDd/GfgWVmvLXuNKyOiNyJ6ly5dmqFY+ZgOYyBKjlkwm8ccIMxskibssC/p1RHxL5L+dIJTPlzjtW8G1kg6HngUuBh41bj3WBYRW9PNC4G70+fXA/+7LDF9HvCuGu/XUjsOlCbqa30N4tgFPVx/5wAR0dIut2bW3qqN6Jqb/p3fyAtHxIikN5Hc7Aska1rfKekKYENErAfeIulCYATYBfx+eu0uSe8nCTIAV0TErkbKMVV2pjWIpdOiBtHD0Mgouw4OTYsajZm1p2oLBv1T+vd9jb54RFxL0vOpfN97yp6/iwlqBhGxDljX6HtPtZ0Hp08NYtmCHgC27h1wgDCzhmUZB7FC0tckbZP0uKSvSFoxFYVrJ6UaxJHTIEm9bMFsIAkQZmaNytKL6TPAeuBYkq6m30z3WZkdB4aY39NJd2eh1UUZq0E8tre/xSUxs3aWJUAsjYjPRMRI+rgKaF13oWlqx4HpMQYCYMm8bjo75BqEmU1KlgCxQ9KrJRXSx6uBnXkXrN3sPDA0LcZAAHR0iKOP6HGAMLNJyRIg/gD4XeAxYCvwinSflZkuo6hLli3oYaubmMxsEmouXBARD5OMUbAqdh4Y4ldWT48aBMCyhbO5ffOeVhfDzNpYtYFy75noGBAR8f4cytOWiqPBrr7pNeZgmQfLmdkkVWtiOljhAfA64M9zLldb2XVwiIjWrkU93jFHJIPldvd50j4za0y1gXJ/X3ouaT7wJ8BlJNN2//1E1x2Odh5M52GaO31qEMcuTLq6btnTPy3GZphZ+6mapJZ0pKS/Bm4nCSZnRMSfR8S2KSldm9iZzsM0rWoQ6WC5x9yTycwaVC0H8bfAbwNXAidHxIEpK1WbmU4zuZYcOzbdhnsymVljqtUg3k4yevovgS2S9qWP/ZKyrAdx2JiONYjFHixnZpNULQeRZYyEkfxKn9XZwRE9Xa0uyphCOljOTUxm1igHgSa47/EDPGXpPDo6pld30mULetjiJiYza5ADRBPc9/h+TjymoWUzcnXMAtcgzKxxDhCTtLdvmK17B6ZlgEim20gGy5mZ1SvXACHpfEn3Stok6Z0Vjv+ppLsk3S7pu5KOKztWlHRr+lifZzkn475t+wE48ejpGCBmM+jBcmbWoNwChKQC8DHgAuAk4BJJJ4077RagNyJOAb4M/E3Zsf6IOC19TNu5oO55LAkQT52mNQhwV1cza0yeNYgzgU0R8UBEDJGMwL6o/ISI+H5E9KWbNwFtt1LdfY/tZ35359i4g+nkmFKA2OM8hJnVL88AsRx4pGx7c7pvIq8DvlW23SNpg6SbJP3mRBdJWpuet2H79u2TK3ED7n18P089Zv60nBBv1ZFzAPjljoM1zjQze7I8A0SlO2bFbGm6CFEv8Ldlu1dFRC/wKuAjkv5HpWsj4sqI6I2I3qVLp3ahu4jg3semZw8mSAbLrVg0m1se2d3qophZG8ozQGwGVpZtrwC2jD9J0guBdwMXRsRgaX9EbEn/PgD8ADg9x7I2ZNv+Qfb2D0/LBHXJGasW8fOHvC6EmdUvzwBxM7BG0vGSZgEXA0/ojSTpdOCfSILDtrL9iyR1p8+XAOcAd+VY1obcW0pQT+sAsZDH9g2wZY8T1WZWn9wCRESMAG8CrgfuBr4UEXdKur6Fug8AAAvmSURBVEJSqVfS3wLzgH8b15316cAGSbcB3wc+GBHTNkBM1yYmgDOOWwTAzx92M5OZ1afmkqOTERHXAteO2/eesucvnOC6G4GT8yxbM9z7+H6Wzu+e1ustPH3ZEXR3dnDLw3t46SnHtro4ZtZGPJJ6Eu59bP+0zj8AdBU6OGXFAtcgzKxuDhANKo4G92+bvj2Yyp2xahF3PrqPwZFiq4tiZm3EAaJBj+zqY2B4dNrXIABOX7WIoeIodzzqZTzMLDsHiAb95L93AHDyigUtLkltZxy3EIBb3MxkZnVwgGjQ+lu3cMLSuTytDZqYjprfkwyYe9jjIcwsOweIBjy2d4D/enAXF5567LScYqOS01ctcqLazOriANGAf799CxFw4ant0230WasWsnXvAA9sP9DqophZm3CAaMA3b9vCM5cfwQlL57W6KJm9+ORldBXEZ3/6UKuLYmZtwgGiTg/tPMhtm/fysjYbdHbUET287NRj+dKGR9jb7wWEzKw2B4g6ffO2ZL7Bl7ZR81LJ655zPH1DRb7wXw+3uihm1gYcIOowOhp849Yt/MrqRSxfOLvVxanbM45dwLNPWMzVNz7IcHG01cUxs2nOAaIOV//0Qe7fdoBLzlzV6qI07PXPPZ4tewf41h2PtbooZjbNOUBktGnbfj74rXt4wdOO4rdOr7Yw3vR27olHccKSuXzse5voH/LUG2Y2MQeIDIaLo7zti7cxZ1aBD7785LYZ+1BJR4d490uezv3b9vPma37OiJuazGwCDhA1jBRH+et/v4tfPLqX//PbJ3PU/J5WF2nSfv3pR/O+C5/Bd+7exl99404iKq4Ea2aHuVzXg2h3m3f38bYv3srND+7m9391Nec/c1mri9Q0r3n2arbuHeDjP/hvRkeDP7/gadN6XQszm3q5BghJ5wP/FygAn4qID4473g18FngWsBN4ZUQ8mB57F/A6oAi8JSKuz7Os5R7Z1cfXb3mUK3/8ABHwD688ld86fcVUvf2UeceLTqQ4GnzqP3/JtXds5c0veAq/27uShXMcKMwMlFfzgqQCcB/wG8BmkjWqLylfOlTSHwOnRMQbJF0M/FZEvFLSScA1wJnAscB3gKdGRNWsam9vb2zYsKGuco6OBndt3cc9j+3nnq37uOWRPWx8KJmz6LlrlvCB3zyZVYvn1PWa7eb+x/fzgWvv5gf3bqdDcOrKhTznKUt4ylHzOG7xXI5d0MOCOV10dxZaXVQzazJJGyOit9KxPGsQZwKbIuKBtBBfAC4CyteWvgi4PH3+ZeCjSjLAFwFfiIhB4JeSNqWv99NmF3I0gpd/4kYGR0bp7uzgacfM5x0vOpGLTjuWFYtmdmAoWXP0fK667Exue2QP37tnGz+8bzsf/f4mxv92mDOrwJxZnfR0ddDd2UFHGyfrzWaSRXNm8aU3PLvpr5tngFgOPFK2vRk4a6JzImJE0l5gcbr/pnHXVuxbKmktsBZg1ar6xyd0Fjr459f2snzRbFYvnkuh4/C96Z26ciGnrlzI237jqQwMF9m8u48Hd/Sxdd8A+/qH2X1wiL7hIgPDRQaHRwmc3DabDo7o6crldfMMEJXutOPvKBOdk+XaZGfElcCVkDQx1VPAkuc9dWkjl81oPV0FnnLUfJ5y1PRf78LM8pFnN9fNwMqy7RXAlonOkdQJLAB2ZbzWzMxylGeAuBlYI+l4SbOAi4H1485ZD1yaPn8F8L1IsubrgYsldUs6HlgD/FeOZTUzs3Fya2JKcwpvAq4n6ea6LiLulHQFsCEi1gOfBj6XJqF3kQQR0vO+RJLQHgHeWKsHk5mZNVdu3VxboZFurmZmh7Nq3Vw91YaZmVXkAGFmZhU5QJiZWUUOEGZmVtGMSlJL2g48VMclS4AdORWn1fzZ2tNM/Wwz9XNB+3+24yKi4mjhGRUg6iVpw0TZ+3bnz9aeZupnm6mfC2b2Z3MTk5mZVeQAYWZmFR3uAeLKVhcgR/5s7WmmfraZ+rlgBn+2wzoHYWZmEzvcaxBmZjYBBwgzM6vosA0Qks6XdK+kTZLe2eryNErSSknfl3S3pDsl/Um6/0hJN0i6P/27qNVlbZSkgqRbJP17un28pJ+ln+2L6XTybUfSQklflnRP+v09e6Z8b5Lelv73eIekayT1tOv3JmmdpG2S7ijbV/F7UuIf0/vK7ZLOaF3JJ++wDBCSCsDHgAuAk4BLJJ3U2lI1bAR4e0Q8HTgbeGP6Wd4JfDci1gDfTbfb1Z8Ad5dtfwj4h/Sz7QZe15JSTd7/Ba6LiKcBp5J8xrb/3iQtB94C9EbEM0mm+7+Y9v3ergLOH7dvou/pApL1a9aQLIX8iSkqYy4OywABnAlsiogHImII+AJwUYvL1JCI2BoRP0+f7ye5ySwn+TxXp6ddDfxma0o4OZJWAC8BPpVuC3gB8OX0lLb8bJKOAJ5HsiYKETEUEXuYId8byVozs9OVIucAW2nT7y0ifkSyXk25ib6ni4DPRuImYKGkZVNT0uY7XAPEcuCRsu3N6b62Jmk1cDrwM+DoiNgKSRABjmpdySblI8CfAaPp9mJgT0SMpNvt+t2dAGwHPpM2n31K0lxmwPcWEY8Cfwc8TBIY9gIbmRnfW8lE39OMurccrgFCFfa1dX9fSfOArwBvjYh9rS5PM0h6KbAtIjaW765wajt+d53AGcAnIuJ04CBt2JxUSdoefxFwPHAsMJek6WW8dvzeapkp/30Ch2+A2AysLNteAWxpUVkmTVIXSXD414j4arr78VLVNv27rVXlm4RzgAslPUjSDPgCkhrFwrTpAtr3u9sMbI6In6XbXyYJGDPhe3sh8MuI2B4Rw8BXgV9lZnxvJRN9TzPq3nK4BoibgTVpr4pZJAm09S0uU0PSNvlPA3dHxIfLDq0HLk2fXwp8Y6rLNlkR8a6IWBERq0m+o+9FxO8B3wdekZ7Wrp/tMeARSSemu36dZA32tv/eSJqWzpY0J/3vs/TZ2v57KzPR97QeeG3am+lsYG+pKaodHbYjqSW9mOTXaAFYFxEfaHGRGiLpOcCPgV9wqJ3+L0jyEF8CVpH8D/s7ETE+0dY2JD0f+F8R8VJJJ5DUKI4EbgFeHRGDrSxfIySdRpJ8nwU8AFxG8qOt7b83Se8DXknSy+4W4PUkbfFt971JugZ4Psm03o8D7wW+ToXvKQ2IHyXp9dQHXBYRG1pR7mY4bAOEmZlVd7g2MZmZWQ0OEGZmVpEDhJmZVeQAYWZmFTlAmJlZRQ4QZoCkoqRbyx5VRzVLeoOk1zbhfR+UtKSB614k6XJJiyRdO9lymFXSWfsUs8NCf0SclvXkiPhknoXJ4LkkA8+eB/ykxWWxGcoBwqyKdJqPLwLnprteFRGbJF0OHIiIv5P0FuANJIPC7oqIiyUdCawjmZSvD1gbEbdLWgxcAywF/ouyuXskvZpkmuxZJAMd/zgiiuPK80rgXenrXgQcDeyTdFZEXJjHv4EdvtzEZJaYPa6J6ZVlx/ZFxJkkI2Q/UuHadwKnR8QpJIEC4H3ALem+vwA+m+5/L/Cf6QR960lG4iLp6SQjj89JazJF4PfGv1FEfJFkzqY7IuJk4I70vR0crOlcgzBLVGtiuqbs7z9UOH478K+Svk4yBQPAc4CXA0TE9yQtlrSApEnot9P9/yFpd3r+rwPPAm5OZmtgNhNP1LcG+O/0+Zx0HRCzpnOAMKstJnhe8hKSG/+FwF9JegbVp32u9BoCro6Id1UriKQNJHMCdUq6C1gm6VbgzRHx4+ofw6w+bmIyq+2VZX9/Wn5AUgewMiK+T7Kw0UJgHvAj0iaidKLBHek6HeX7LwBKa05/F3iFpKPSY0dKOm58QSKiF/gPkvzD3wDvjojTHBwsD65BmCVmp7/ES66LiFJX125JPyP5QXXJuOsKwL+kzUciWXN5T5rE/oyk20mS1KWpod8HXCPp58APSWYCJSLukvSXwLfToDMMvBF4qEJZzyBJZv8x8OEKx82awrO5mlWR9mLqjYgdrS6L2VRzE5OZmVXkGoSZmVXkGoSZmVXkAGFmZhU5QJiZWUUOEGZmVpEDhJmZVfT/Af2Yo7YKuRw6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax2 = fig.add_subplot(121)\n",
    "plt.plot(np.arange(1, len(arr_noise)+1), arr_noise)\n",
    "plt.ylabel('Noise Scale, gamma=0.99')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Watch a Smart Agent!\n",
    "\n",
    "Each step of the Smart Agent animation includes the cl of the following _action_, _state_ and _reward_.     \n",
    "This also includes a slight _delay_ (0.1 sec) for better observation of the animation, and  further rendering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "timesteps:  500\n",
      "time :  27.902000188827515\n"
     ]
    }
   ],
   "source": [
    "t1 = time.time()\n",
    "\n",
    "state = env.reset()\n",
    "timesteps = 0\n",
    "while True:\n",
    "    action = policy.act(state)\n",
    "    env.render()\n",
    "    time.sleep(0.05)\n",
    "    timesteps += 1\n",
    "    state, reward, done, _ = env.step(action)\n",
    "    if done:\n",
    "        break \n",
    "\n",
    "print('timesteps: ', timesteps)\n",
    "delta = time.time() - t1\n",
    "print('time : ', delta)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
